Skip to content

Numpy Datatype: Enable np.dtype() type conversions - #1856

Open
franzpoeschel wants to merge 9 commits into
openPMD:devfrom
franzpoeschel:dataset-constructor-conversions
Open

Numpy Datatype: Enable np.dtype() type conversions#1856
franzpoeschel wants to merge 9 commits into
openPMD:devfrom
franzpoeschel:dataset-constructor-conversions

Conversation

@franzpoeschel

@franzpoeschel franzpoeschel commented Mar 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

This makes it possible to use things like np.int_ or "double" in the Dataset constructor, which currently has to be written as np.dtype(np.int_) and np.dtype("double"). Check also examples and tests.
Affects Datatype() constructor as well as RecordComponent.make_empty().

Future: Derive Dataset definition from Numpy array directly

@franzpoeschel franzpoeschel mentioned this pull request Mar 2, 2026
Merged
1 task
github-advanced-security[bot]
github-advanced-security AI found potential problems Mar 2, 2026
View reviewed changes
Comment thread test/python/unittest/API/APITest.py Fixed
Comment thread test/python/unittest/API/APITest.py Fixed
Comment thread test/python/unittest/API/APITest.py Fixed
Comment thread test/python/unittest/API/APITest.py Fixed
Comment thread test/python/unittest/API/APITest.py Fixed
Comment thread test/python/unittest/API/APITest.py Fixed
Comment thread test/python/unittest/API/APITest.py Fixed
Comment thread test/python/unittest/API/APITest.py Fixed
Comment thread test/python/unittest/API/APITest.py Fixed
Comment thread test/python/unittest/API/APITest.py Fixed
@franzpoeschel
franzpoeschel force-pushed the dataset-constructor-conversions branch 2 times, most recently from 3d996b7 to a4c6bf9 Compare March 2, 2026 18:23
@franzpoeschel
franzpoeschel requested a review from ax3l March 3, 2026 10:21
ax3l
ax3l reviewed Mar 3, 2026
View reviewed changes
Comment thread examples/10_streaming_write.py Outdated
ax3l
ax3l reviewed Mar 3, 2026
View reviewed changes
Comment thread include/openPMD/binding/python/Numpy.hpp Outdated
Comment on lines +123 to +127
try
{
create_dtype = numpy.attr("dtype");
}
catch (std::exception const &e)

@ax3l ax3l Mar 3, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe check if it has the attribute and then throw? Avoids one catch

@franzpoeschel franzpoeschel Mar 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have removed those try-catch blocks which should only throw if the numpy interfaces change. This leaves those two blocks that check if numpy is installed and for any errors when calling numpy.dtype() with user-supplied objects.

franzpoeschel
franzpoeschel commented Mar 3, 2026
View reviewed changes
Comment thread include/openPMD/binding/python/Numpy.hpp Outdated
franzpoeschel
franzpoeschel commented Mar 3, 2026
View reviewed changes
Comment thread src/binding/python/Dataset.cpp Outdated
@franzpoeschel
franzpoeschel force-pushed the dataset-constructor-conversions branch from 752d517 to 6c346d0 Compare March 4, 2026 12:31
@franzpoeschel
franzpoeschel requested a review from ax3l March 4, 2026 16:12
@franzpoeschel
franzpoeschel force-pushed the dataset-constructor-conversions branch from 6c346d0 to b164b88 Compare March 20, 2026 11:32
@franzpoeschel
franzpoeschel force-pushed the dataset-constructor-conversions branch from b164b88 to 3f96ee9 Compare June 10, 2026 09:17
@franzpoeschel
franzpoeschel force-pushed the dataset-constructor-conversions branch from 3f96ee9 to 037e8a2 Compare August 25, 2026 11:22
@franzpoeschel franzpoeschel mentioned this pull request Aug 25, 2026
Open
github-advanced-security[bot]
github-advanced-security AI found potential problems Aug 25, 2026
View reviewed changes
Comment thread test/python/unittest/API/APITest.py
self.assertEqual(dc128.dtype, "complex128")
if file_ending not in ["bp", "bp4", "bp5"]:
self.assertTrue(dc256.dtype == np.dtype("clongdouble"))
self.assertTrue(dc256.dtype == "clongdouble")

Check failure

Code scanning / CodeQL

Potentially uninitialized local variable Error test

Local variable 'dc256' may be used before it is initialized.
Comment thread test/python/unittest/API/APITest.py
self.assertTrue(ms["uint64"][SCALAR].constant)
self.assertTrue(ms["double"][SCALAR].constant)

self.assertTrue(ms["int16"][SCALAR].load_chunk(o, e).dtype == "int16")

Check notice

Code scanning / CodeQL

Imprecise assert Note test

assertTrue(a == b) cannot provide an informative message. Using assertEqual(a, b) instead will give more informative messages.
Comment thread test/python/unittest/API/APITest.py
self.assertTrue(ms["double"][SCALAR].constant)

self.assertTrue(ms["int16"][SCALAR].load_chunk(o, e).dtype == "int16")
self.assertTrue(ms["int32"][SCALAR].load_chunk(o, e).dtype == "int32")

Check notice

Code scanning / CodeQL

Imprecise assert Note test

assertTrue(a == b) cannot provide an informative message. Using assertEqual(a, b) instead will give more informative messages.
Comment thread test/python/unittest/API/APITest.py

self.assertTrue(ms["int16"][SCALAR].load_chunk(o, e).dtype == "int16")
self.assertTrue(ms["int32"][SCALAR].load_chunk(o, e).dtype == "int32")
self.assertTrue(ms["int64"][SCALAR].load_chunk(o, e).dtype == "int64")

Check notice

Code scanning / CodeQL

Imprecise assert Note test

assertTrue(a == b) cannot provide an informative message. Using assertEqual(a, b) instead will give more informative messages.
Comment thread test/python/unittest/API/APITest.py
self.assertTrue(ms["int16"][SCALAR].load_chunk(o, e).dtype == "int16")
self.assertTrue(ms["int32"][SCALAR].load_chunk(o, e).dtype == "int32")
self.assertTrue(ms["int64"][SCALAR].load_chunk(o, e).dtype == "int64")
self.assertTrue(ms["uint16"][SCALAR].load_chunk(o, e).dtype == "uint16")

Check notice

Code scanning / CodeQL

Imprecise assert Note test

assertTrue(a == b) cannot provide an informative message. Using assertEqual(a, b) instead will give more informative messages.
Comment thread test/python/unittest/API/APITest.py
self.assertTrue(ms["int32"][SCALAR].load_chunk(o, e).dtype == "int32")
self.assertTrue(ms["int64"][SCALAR].load_chunk(o, e).dtype == "int64")
self.assertTrue(ms["uint16"][SCALAR].load_chunk(o, e).dtype == "uint16")
self.assertTrue(ms["uint32"][SCALAR].load_chunk(o, e).dtype == "uint32")

Check notice

Code scanning / CodeQL

Imprecise assert Note test

assertTrue(a == b) cannot provide an informative message. Using assertEqual(a, b) instead will give more informative messages.
Comment thread test/python/unittest/API/APITest.py
self.assertTrue(ms["int64"][SCALAR].load_chunk(o, e).dtype == "int64")
self.assertTrue(ms["uint16"][SCALAR].load_chunk(o, e).dtype == "uint16")
self.assertTrue(ms["uint32"][SCALAR].load_chunk(o, e).dtype == "uint32")
self.assertTrue(ms["uint64"][SCALAR].load_chunk(o, e).dtype == "uint64")

Check notice

Code scanning / CodeQL

Imprecise assert Note test

assertTrue(a == b) cannot provide an informative message. Using assertEqual(a, b) instead will give more informative messages.
Comment thread test/python/unittest/API/APITest.py
self.assertTrue(ms["uint16"][SCALAR].load_chunk(o, e).dtype == "uint16")
self.assertTrue(ms["uint32"][SCALAR].load_chunk(o, e).dtype == "uint32")
self.assertTrue(ms["uint64"][SCALAR].load_chunk(o, e).dtype == "uint64")
self.assertTrue(ms["single"][SCALAR].load_chunk(o, e).dtype == "single")

Check notice

Code scanning / CodeQL

Imprecise assert Note test

assertTrue(a == b) cannot provide an informative message. Using assertEqual(a, b) instead will give more informative messages.
Comment thread test/python/unittest/API/APITest.py
self.assertTrue(ms["uint32"][SCALAR].load_chunk(o, e).dtype == "uint32")
self.assertTrue(ms["uint64"][SCALAR].load_chunk(o, e).dtype == "uint64")
self.assertTrue(ms["single"][SCALAR].load_chunk(o, e).dtype == "single")
self.assertTrue(ms["double"][SCALAR].load_chunk(o, e).dtype == "double")

Check notice

Code scanning / CodeQL

Imprecise assert Note test

assertTrue(a == b) cannot provide an informative message. Using assertEqual(a, b) instead will give more informative messages.
Comment thread test/python/unittest/API/APITest.py
self.assertEqual(dc128.dtype, "complex128")
if file_ending not in ["bp", "bp4", "bp5"]:
self.assertTrue(dc256.dtype == np.dtype("clongdouble"))
self.assertTrue(dc256.dtype == "clongdouble")

Check notice

Code scanning / CodeQL

Imprecise assert Note test

assertTrue(a == b) cannot provide an informative message. Using assertEqual(a, b) instead will give more informative messages.
@ax3l ax3l self-assigned this Aug 25, 2026
@ax3l ax3l added this to the 0.18.0 milestone Aug 25, 2026
ax3l
ax3l approved these changes Aug 25, 2026
View reviewed changes

@ax3l ax3l left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - looks bw compatible. Please fix the one CodeQL error it brought up and then ready :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ax3l ax3l ax3l approved these changes

Assignees

@ax3l ax3l

Labels

api: new additions to the API frontend: Python3

Projects

None yet

Milestone

0.18.0

Development

Successfully merging this pull request may close these issues.

3 participants

@franzpoeschel @ax3l @github-advanced-security